United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
I nihil Stall-, Patent ami Trademark Office 

Address: COMMISSIONER FOR PATENTS 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. CONFIRMATION NO. 



10/521.732 



01/19/2005 



23117 7590 06/15/2009 

NIXON & VANDERHYE, PC 

901 NORTH GLEBE ROAD, 1 1TH FLOOR 

ARLINGTON, VA 22203 



OLUDE AFOLABI, OLATOYOSI 



PAPER NUMBER 



DELIVERY MODE 



Please find below and/or attached an Office communication concerning this application or proceeding. 

The time period for reply, if any, is set in the attached communication. 



PTOL-90A (Rev. 04/07) 



l/ffflrC? nVrliUli Otfff Iff ids y 


Application No. 

10/521,732 


Applicant(s) 

XU ET AL. 


Examiner 

Ola Olude-Afolabi 


Art Unit 

2129 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address — 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )KI Responsive to communication(s) filed on 05/05/09 . 
2a )^ This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 11-18 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) |EI Claim(s) 11-18 is/are rejected. 

7) 0 Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) Q The specification is objected to by the Examiner. 

10) ^ The drawing(s) filed on 10/30/06 is/are: a)^ accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

20 Certified copies of the priority documents have been received in Application No. . 

3.Q Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attach ment(s) 

1) D Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-41 3) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) Paper No(s)/Mail Date. . 

3) Information Disclosure Statement(s) (PTO/SB/08) 5 ) □ Notice of Informal Patent Application 
Paper No(s)/Mail Date 5/22/09 . 6) □ Other: . 



PTOL-T26 d (Rev e 08-06r 



Office Action Summary 



Part of Paper No./Mail Date 20090527 



Application/Control Number: 10/521 ,732 Page 2 

Art Unit: 2129 

DETAILED ACTION 

1. This communication is considered fully responsive to the 
Amendment filed on 5/05/2009 for the patent application 
10/521,732. Claims 11-18 have been examined and remain 
pending . 



Claim Rejections - 35 USC § 103 



2. The following is a quotation of 35 U.S.C. 103(a) which forms 
the basis for all obviousness rejections set forth in this 
Office action: 

(a) A patent may not be obtained though the invention is not identically 
disclosed or described as set forth in section 102 of this title, if the 
differences between the subject matter sought to be patented and the prior 
art are such that the subject matter as a whole would have been obvious at 
the time the invention was made to a person having ordinary skill in the 
art to which said subject matter pertains. Patentability shall not be 
negatived by the manner in which the invention was made. 



3. Claims 11-18 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over C. C. Chibelushi et al . (A .Review of Speech- 
Based Bimodal Recognition) (Chibelushi hereafter) and Haykin 
(Neural Networks : A Comprehensive Foundation , Chapter 9) 



Regarding Claim 11, and 15: 
Chibelushi discloses: 

feature extraction means (e.g. Mouth-window methods, col. 
1: par. 2a, pp. 2 6, Chibelushi) for extracting by a feature 
extraction module (e.g. MFCC feature extraction, Fig. 2) a 
plurality of sets of characteristic visual feature vectors 
and a plurality of sets of characteristic audio feature 
vectors from respective video and audio portions of a 
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training set comprising a plurality of video sequences 
belonging to a predetermined class; 

feature combining means (e.g. sensor fusion, Low-level 
fusion can occur at the data level or feature level. 
Intermediate-level and high-level fusion involves the 
combination of recognition scores or labels produced as 
intermediate or final output of classifiers, col. 1 Section 
B pp. 28, Chibelushi) for combining by a feature binder the 
plurality of sets of characteristic visual and audio 
feature vectors into a respective plurality of N- 
dimensional feature vectors corresponding to the 
predetermined class (e.g. audio-visual fusion can also 
occur at a level between feature and decision levels, fig. 
1, Chibelushi), said combining comprising normalizing and 
concatenating each of the visual feature vectors with 
corresponding audio feature vectors (e.g. Some similarity 
measures are tightly coupled to particular feature types. 
For speaker verification or open-set identification, a 
normalization of similarity scores may be necessitated by 
speech variability [63], [101]. Examples of common 
similarity measures are: the Euclidean distance (often 
inverse-variance weighted, or reduced to a city-block 
distance), col. 2, section C, pp. 27- col. 1: 1-8, pp. 28, 
Chibelushi) ; 

analysing by a feature learning module the pluralities of 
N-dimensional feature vectors using principal component 
analysis (e.g. Some high-level features aim at reducing 
dimensionality through a transformation (e.g. transforms 
are based on principal component analysis (PCA) , 
statistical discriminant analysis optimizing the F-ratio 
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such as linear discriminant analysis (LDA) [1], and 
integrated mel-scale representation with LDA ( IMELDA) ) that 
produces statistically orthogonal features and packs most 
of the variance into few features, col. 1, pp. 25, 
Chibelushi) or kernel discriminant analysis to generate a 
set of M basis vectors, each being of N-dimensions (e.g. 
applied to static spectral information, possibly combined 
with dynamic spectral information, output by a mel-scale 
filter bank. Composite features are sometimes generated by 
a simple concatenation of different types of features, col. 
2 pp. 25, Chibelushi); 

Chibelushi fails to particularly call for: 

- a plurality of sets of characteristic visual feature 
vectors and a plurality of sets of characteristic audio 
feature vectors from respective video and audio portions of 
a training set comprising a plurality of video sequences 
wherein M << N, and using the set of M basis vectors, 
mapping each N-dimensional feature vector into a respective 
M-dimensional feature vector 

using the M-dimensional feature vectors thus obtained as 
the basis for or as input to train a class model of the 
predetermined class 

storing the class model for use in classifying input data 
that matches the predetermined class 

Haykin teaches: 

a plurality of sets of characteristic visual feature 
vectors and a plurality of sets of characteristic audio 
feature vectors from respective video and audio portions of 
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a training set (e.g. input pattern presented to the 
network, (pp. 443-483 especially pp.447, Haykin) comprising 
a plurality of video sequences (e.g. data from input space, 
fig. 9.4 pp. 455, Haykin). 

wherein M << N, and using the set of M basis vectors, 
mapping each N-dimensional feature vector into a respective 
M-dimensional feature vector (Examiner's Note: continuous 
input space is mapped to discrete output space through the 
feature map, fig. 9.4 pp. 455, Haykin); 

using the M-dimensional feature vectors thus obtained as 
the basis for or as input to train a class model of the 
predetermined class (e.g. input pattern presented to the 
network, (pp. 443-483 especially pp.447, Haykin). 
storing the class model for use in classifying input data 
that matches the predetermined class (Examiner's Note: the 
aim of the SOM algorithm is to store a large set of input 
vectors by finding a smaller set of prototypes, so as to 
provide a good approximation to the original input space, 
see par. 2, pp. 455, Haykin) . 
Rationale : 

Thus, it would have been recognized by one of ordinary 
skill in the art at the time of the invention to modify the 
teachings of Chibelushi for generating class models from 
video sequences having one of a plurality of predetermined 
classes with the teachings of Haykin for the benefit of 
reducing dimensionality or compressing data and also to 
store a large set of input vectors by finding a smaller set 
of prototypes, so as to provide a good approximation to the 
original input space. 
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Regarding Claims 12 and 16, The computer-implemented method as 
claimed in claim 11, wherein the M basis vectors are the M most 
discriminating basis vectors that maximize between-class 
variance and minimize within-class variance (e.g. conditions for 
the minimization of the expected distortion which is given the 
input x, choose the code c = c (x) to minimize the squared error 
distortion | |x - x' | |2, pp. 456, Haykin) . 

Regarding Claims 13 and 17, The computer-implemented method as 
claimed in claim 11 wherein each video sequence has a non-linear 
feature distribution (e.g. video sequence is a feature map, 
(Fig. 9.7b pp. 462), and the property 4 of a feature map is 
feature selection, see pp. 461, Haykin) . 

Regarding Claims 14 and 18, The computer-implemented method as 
claimed in claim 12 wherein each video sequence has a non-linear 
feature distribution (e.g. video sequence is a feature map, 
(Fig. 9.7b pp. 462), and the property 4 of a feature map is 
feature selection i.e. video sequence is data from an input 
space with a non-linear distribution, pp. 461, Haykin) . 

The prior art made of record and not relied upon is considered 

pertinent to applicant's disclosure. 

Tsoukalas et al . , discloses 

feature vectors are corresponding inputs as the 
components of an m-dimensional input vector 
(Tsoukalas: pp. 309). 
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kernel discriminant analysis to generate a set of M 
basis vectors sometimes referred to as m-dimensional 
input vector (Tsoukalas : pp. 309) . 

a feature learning module the pluralities of N- 
dimensional feature vectors (sometimes called n- 
dimensional weight vector (Tsoukalas : pp. 309) . 

a variation of the learning scheme of neurons, which 
makes the Kohonen network into a classification 
network called Learning Vector Quantization (LVQ) , the 
modification involves changing the training scheme, 
which requires a collection of training examples each 
assigned to one of a set of known classes {Tsoukalas : 
pp. 314). 

Sudaram. , discloses the objective function that is to be 
minimized has constraints (audio/video duration 

constraints, visual syntax, synchronous multimedia 
constraints) that are constructed with the aim of 
maximizing the speech information content and the overall 
coherence of the video (Sudaram: col. 1, par. 3, pp. 2). 



RESPONSE TO ARGUMENTS 

Applicants' arguments filed April 07, 2009 have been fully 
considered but are not found persuasive. Specifically, 
Applicants made the following arguments: 



1 . Rejection Based on U.S.C. 103 : 



Argument 1 
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In the Office Action, the Examiner has rejected 
claims 11-18 under 35 U.S.C. § 103(a) as being 
unpatentable over Chibelushi et al . ( "Chibelushi " ) 
and Haykin. Applicants respectfully traverse the 
Examiner's § 103 rejections of the claims. 

The Examiner has not met the burden of providing a 
prima facie case of obviousness based on the above- 
described references. Applicants have not been 
provided with a copy of the relevant portions of the 
Haykin reference nor have Applicants been provided 
with copies or even an identification of the 
Tsoukalas reference which is cited by the Examiner at 
pages 3 and 4 of the Office Action (Emphasis Added) . 



Applicants argues that a copy of the relevant portions of the 
Haykin reference nor have Applicants been provided with copies 
or even an identification of the Tsoukalas reference which is 
cited by the Examiner at pages 3 and 4 of the Office Action has 
not been provided. Examiner disagrees. 

First , an identification of the Haykin reference was provided 
to the applicants as a Non-Patent Literature. If a copy of the 
relevant portions of the Haykin reference was needed, applicant 
should have requested those portions within 1 month of the mail 
date of the action (see MPEP 710.06), however, applicant did not 
request these relevant portions until 6 months after the mailing 
date of the action. 

Second , the Tsoukalas reference was not provided because the 
claims were not rejected by Tsoukalas. The Tsoukalas and 
Sudaram references are given in Notes/Examiner's Notes, which 
are not prior art but a link to prior art that one of ordinary 
skill in the art would find relevant. 

Examiner's Notes are/if provided with the cited references to 
prior art to assist the applicant to better understand the 
nature of the prior art, application of such prior art and, as 
appropriate, to further indicate other prior art that maybe 
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applied in other office actions. Such comments are entirely 
consistent with the intent and spirit of compact prosecution. 
However, and unless otherwise stated, the citations are self- 
explanatory to one skilled in the art and do not need any 
further explanation. 



Argument 2 

Accordingly, the Examiner's § 103 rejection of the 
claims is defective in that Applicants have not been 
provided the information needed to rebut the 
Examiner's assertions. Therefore, it is respectfully 
submitted that the Examiner's rejections of the 
claims are improper and must be withdrawn. 

Therefore, in view of the above remarks, it is 
respectfully requested that the Examiner issue a new 
non-final Office Action including citations to any 
prior art relied upon as well as providing copies of 
the prior art relied upon (Emphasis Added) . 

Applicants argues that the Examiner's rejections 
of the claims are improper and must be withdrawn and 
issue a new non-final Office Action including 
citations to any prior art relied upon as well as 
providing copies of the prior art relied upon. 
Examiner strongly disagrees. 

In response, applicant's claims were rejected under 35 
U.S.C. 103(a) as being unpatentable over C. C. Chibelushi et al . 
(A Review of Speech-Based Bimodal Recognition) (Chibelushi 
hereafter) and Haykin (Neural Networks: A Comprehensive 
Foundation, Chapter 9) . 

In this case, the cited references were correct and both 
prior arts were made available to the applicant and there was no 
error in the Office action. 



Accordingly, the rejections of the claims STAND. 
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Claim Interpretation 

a. The claims and only the claims form the metes and 
bounds of the invention. "Office personnel are to give the 
claims their broadest reasonable interpretation in light of 
the supporting disclosure. In re Morris, 127 F.3d 1048, 
1054-55, 44USPQ2d 1023, 1027-28 (Fed. Cir. 1997) . 
Moreover, limitations appearing in the specification but 
not recited in the claim are not read into the claim. In re 
Prater, 415 F.2d, 1393, 1404-05, 162 USPQ 541,550-551 (CCPA 
1969)" (MPEP p 2100-8, c 2,145-48; p 2100-9, c 1,1 1-4). 

The Examiner has full latitude to interpret each claim in 
the broadest reasonable sense. The Examiner will reference 
prior art using terminology familiar to one of ordinary skill in 
the art. Such an approach is broad in concept and can be either 
explicit or implicit in meaning. 

Examiner's Notes are/if provided with the cited references 
to prior art to assist the applicant to better understand the 
nature of the prior art, application of such prior art and, as 
appropriate, to further indicate other prior art that maybe 
applied in other office actions. Such comments are entirely 
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consistent with the intent and spirit of compact prosecution. 
However, and unless otherwise stated, the citations are self- 
explanatory to one skilled in the art and do not need any 
further explanation. Moreover, the Examiner's Notes are not 
prior art but a link to prior art that one of ordinary skill in 
the art would find inherently or obviously appropriate. 

Unless otherwise annotated, as aforementioned, Examiner's 
statements are to be interpreted in reference to that of one of 
ordinary skill in the art. Statements made in reference to the 
condition of the disclosure constitute, on the face of it, the 
basis and such would be obvious to one of ordinary skill in the 
art, establishing thereby an inherent or obviousness prima facie 
case or statement ( s ) . 

EXAMINER'S NOTE 

Examiner has cited particular columns and line numbers or 
paragraph numbers in the references applied to the claims above 
for the convenience of the applicant. Although the specified 
citations are representative of the teachings of the art and are 
applied to specific limitations within the individual claim, 
other passages and figures may apply as well. It is 
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respectfully requested from the applicant in preparing 
responses, to fully consider the references in their entirety as 
potentially teaching all or part of the claimed invention, as 
well as the context of the passage as taught by the prior art or 
disclosed by the Examiner. The entire reference is considered 
to provide disclosure relating to the claimed invention. 

In the case of amending the claimed invention, Applicant is 
respectfully requested to indicate the portion (s) of the 
specification which dictate (s) the structure relied on for 
proper interpretation and also to verify and ascertain the metes 
and bounds of the claimed invention. 

Conclusion 

Applicant's amendment necessitated the new ground (s) of 
rejection presented in this Office action. Accordingly, THIS 
ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is 
reminded of the extension of time policy as set forth in 37 
CFR 1 . 136 (a) . 

A shortened statutory period for reply to this final action 
is set to expire THREE MONTHS from the mailing date of this 
action. In the event a first reply is filed within TWO MONTHS 
of the mailing date of this final action and the advisory action 
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is not mailed until after the end of the THREE-MONTH shortened 
statutory period, then the shortened statutory period will 
expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated 
from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than 
SIX MONTHS from the date of this final action. 
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Correspondence In forma tl on 

Any inquires concerning this communication or earlier 
communications from the examiner should be directed to Ola 
Olude-Af olabi , who may be reached Monday through Friday, between 
8:00 a.m. and 5:00 p.m. EST. or via telephone at (571) 270-5639 
or facsimile transmission (571) 270-6639. 

If you need to send an Official facsimile transmission, 
please send it to (571) 273-8300 . If attempts to reach the 
examiner are unsuccessful the Examiner's Supervisor, David 
Vincent, may be reached at (571) 272-3080. 

Hand-delivered responses should be delivered to the 
Receptionist @ (Customer Service Window Randolph Building 401 
Dulany Street Alexandria, VA 22313), located on the first floor 
of the south side of the Randolph Building. 

Finally, information regarding the status of an application 
may be obtained from the Patent Application Information 
Retrieval (PAIR) system. Moreover, status information for 
published applications may be obtained from either Private PAIR 
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or Public PAIR. Status information for unpublished applications 
is available through Private PAIR only. For more information 
about the PAIR system, see http : / /pair-direct . uspto . gov . Should 
you have any questions on access to the Private PAIR system, 
contact the Electronic Business Center (EBC) toll-free @ 1-866- 
217-9197 . 

Ola Olude-Afolabl 

Patent Examiner 
Artificial Intelligence 
United States Patent & 
Trademark Office 

Thursday, June 04, 2009 
/O. 0./ 

Examiner, Art Unit 2129 
/David R Vincent/ 

Supervisory Patent Examiner , Art Unit 2129 



